SYSTEM AND METHOD FOR CONCURRENTLY DEMOSAICING AND 
RESIZING RAW DATA IMAGES 



FIELD OF THE INVENTION 

The invention relates generally to the field of image processing, and more 
particularly to a system and method for demosaicing and resizing raw data 
(mosaiced) images. 

BACKGROUND OF THE INVENTION 

Color digital cameras are becoming ubiquitous in the consumer 
marketplace, partly due to progressive price reductions. Color digital cameras 
typically employ a single optical sensor, either a Charge Coupled Device (CCD) 
sensor or a Complementary Metal Oxide Semiconductor (CMOS) sensor, to 
digitally capture a scene of interest. Both CCD and CMOS sensors are only 
sensitive to light intensity. Consequently, these sensors cannot discriminate 
between different colors. In order to achieve color discrimination, a color filtering 
technique is applied to separate light in terms of base colors, typically red, green 
and blue. 

A common filtering technique utilizes a color-filter array (CFA), which is 
overlaid on a sensor array, to separate colors of impinging light in a Bayer pattern. 
The Bayer pattern is a periodic pattern with a period of two different color pixels 
in each dimension (vertical and horizontal). In the horizontal direction, a single 
period includes either a green pixel and a red pixel, or a blue pixel and a green 
pixel. In the vertical direction, a single period includes either a green pixel and a 
blue pixel, or a red pixel and a green pixel. Therefore, the number of green pixels 
is twice the number of red or blue pixels. The reason for the disparity in the 
number of green pixels is because the human eye is not equally sensitive to all 
three primary colors. Consequently, more green pixels are needed to create a color 
image of a scene that will be perceived as a "true color" image. 
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Due to the CFA, the image captured by the sensor is therefore a mosaiced 
image, also called "raw data" image, where each pixel only holds the value for 
either red, green or blue. The raw data image can then be demosaiced to create a 
color image by estimating the missing color values for each pixel of the image. 

5 These missing color values are estimated by using color information from 
surrounding pixels. 

There are a number of conventional demosaicing methods to convert a raw 
data image into a color image. Three main common categories of demosaicing 
methods include interpolation-based methods, feature-based methods, and 

10 Bayesian methods. The interpolation-based demosaicing methods use simple 
interpolation formulas to interpolate the color planes separately. The 
interpolation-based demosaicing methods include bi-linear methods, band-limited 
interpolation methods using sincQ functions, spline interpolation methods, and the 
like. The feature-based demosaicing methods examine local features of a given 

15 image at the pixel level, and then interpolate the image accordingly. The basic 
idea of the feature-based methods is to avoid interpolating across edges of 
features. The Bayesian methods attempt to find the most probable color image, 
given the data, by assuming some prior knowledge of the image structure. 

After the raw data images have been demosaiced, the images may be 

20 resized for a particular application. As an example, the demosaiced images may 
be reduced to ensure that the images are properly transmitted through a 
communications channel having a predefined bandwidth for video conferencing. 
As another example, the demosaiced images may be reduced to provide thumbnail 
images of the captured images for the user to preview. There are a number of 

25 conventional methods to resize an image into a smaller image. One common 

method involves creating a smaller version of the original image where each pixel 
in the smaller image receives the color values of the closest pixel in the original 
image. Another common method involves low-pass filtering or interpolating the 
original image and then decimating the image at the appropriate rate to produce a 

30 smaller image. The low-pass filtering or interpolation reduces aliasing in the 
decimating step. 
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Although the conventional methods for separately demosaicing raw data 
images and resizing the demosaiced images work well to produce demosaiced and 
resized images, there is a need for a system and method for more efficiently 
demosaicing and resizing raw data images to produce the demosaiced and resized 
5 images. 

SUMMARY OF THE INVENTION 

A system and method for processing mosaiced or raw data images operates 
10 to concurrently demosaic and resize the mosaiced images in a combined process. 
The combined demosaic/resize process allows the system to perform demosaicing 
and resizing more efficiently than conventional systems, which perform these 
processes separately and sequentially. Furthermore, the combined demosaic/resize 
process allows the system to produce demosaiced and resized images of higher 
15 quality as compared to demosaiced and resized images produced by the 
conventional systems. 

A method in accordance with present invention includes receiving a 
mosaiced image to be concurrently demosaiced and resized. The mosaiced image 
is then partitioned into image blocks, which are sequentially processed. For each 
20 image block of the mosaiced image, predefined indicators are computed. In one 
realization, the computed indicators are statistical indicators, such as the variances 
of R, B, Gl and G2 color values within the current image block. In another 
realization, the computed indicators are feature-based indicator, such the gradients 
of R, B and G color values within the current image block. Next, the means of R, 
25 B and G color values within the current image block are computed. 

If any of the computed indicators exceeds a predefined threshold, the 
current image block is divided in half to produce a new current image block, 
which is one of the halves of the current image block. The new current image 
block is then processed in the same manner as the last current image block. 
30 However, if none of the computed indicators exceeds the predefined threshold, the 
computed means of the current image block are embedded into a pixel of the final 
image, which is a demosaiced and resized image of the mosaiced image. 
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A system in accordance with the invention includes an image pipeline unit 
that receives a mosaiced image to be concurrently demosaiced and resized. The 
mosaiced image may be received from an image capturing unit of the system that 
electronically captures a scene of interest as a mosaiced image. The image 

5 pipeline unit includes an image partitioning module that partitions the mosaiced 
image into image blocks. The image pipeline unit also includes an indicator 
computer, processor and a color inserter. For each image block of the mosaiced 
image, predefined indicators are computed by the indicator computer. In one 
realization, the computed indicators are statistical indicators, such as the variances 

10 ofR, B, Gl and G2 color values within the current image block. In another 

realization, the computed indicators are feature-based indicator, such the gradients 
of R, B and G color values within the current image block. In addition, for each 
image block of the mosaiced image, the means of R, B and G color values within 
the current image block are computed by the color inserter. 

15 For a given image block, the processor determines whether any of the 

computed indicators exceeds a predefined threshold. If so, the current image 
block is divided in half to produce a new current image block, which is one of the 
halves of the current image block. The new current image block is then processed 
in the same manner as the last current image block. However, if none of the 

20 computed indicators exceeds the predefined threshold, the computed means of the 
current image block are embedded into a pixel of the final image. 

An advantage of the invention is that the combined demosaic/resize 
process increases the efficiency to demosaic and to resize mosaiced images. In 
addition, the image quality of the resulting demosaiced and resized image can be 

25 significantly superior than demosaiced and resized images produced by 
conventional systems and methods. 

Other aspects and advantages of the present invention will become 
apparent from the following detailed description, taken in conjunction with the 
accompanying drawings, illustrated by way of example of the principles of the 

30 invention. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram of an image processing system in accordance 
with a first embodiment of the present invention. 
5 Fig. 2A illustrates the Bayer pattern of color values in a mosaiced image. 

Fig. 2B illustrates the different color planes of a Bayer-patterned mosaiced 

image. 

Fig. 3 is a process flow diagram of the demosaicing and resizing operation 
of the image processing system of Fig. 1. 
10 Fig. 4A illustrates a partitioned 8x8 image block of a mosaiced image. 

Fig. 4B illustrates an 8x8 image block of a mosaiced image that has been 
vertically divided in half. 

Fig. 4C illustrates an 8x8 image block of a mosaiced image that has been 
horizontally divided in half. 
15 Fig. 4D illustrates a 4x8 image block of a mosaiced image that has been 

horizontally divided in half. 

Fig. 4E illustrates an 8x4 image block of a mosaiced image that has been 
vertically divided in half. 

Fig. 4F illustrates a 2x2 image block of a mosaiced image. 
20 Fig. 5 is a block diagram of an image processing system in accordance 

with a second embodiment of the present invention. 

Fig. 6 is a process flow diagram of the demosaicing and resizing operation 
of the image processing system of Fig. 5. 

25 

DETAILED DESCRIPTION 

With reference to Fig. 1, an image processing system 100 in accordance 
with a first embodiment of the invention is shown. The image processing system 
30 operates to electronically capture a scene of interest as a mosaiced or raw data 
images and then concurrently demosaic and resize the mosaiced image, which 
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increases the efficiency of the system and improves the quality of the resulting 
demosaiced and resized image. 

The image processing system 100 includes an image capturing unit 102, an 
image pipeline unit 104, a storage unit 106 and a viewing device 108. The image 

5 capturing unit 102 of the system operates to electronically capture a scene of 
interest in the form of a mosaiced or raw data image. The image capturing unit 
includes an electronic sensor and a color-filter array (CFA). The electronic sensor 
may be a Charge Coupled Device (CCD) sensor, a Complementary Metal Oxide 
Semiconductor (CMOS) sensor, or other type of photosensitive sensors. In an 

10 exemplary embodiment, the CFA includes red (R), green (G) and blue (B) filters 
arranged in a Bayer filter pattern. However, the CFA may include filters of other 
colors arranged in a different filter pattern. The CFA operates to allow only light 
of a particular color to be transmitted to each photosensitive element of the sensor. 
Thus, a digital image captured by the image capturing unit is a mosaiced image 

1 5 composed of single-colored pixels that are arranged in a color pattern in 

accordance with the filter pattern of the CFA. Consequently, each pixel of the 
mosaiced image has an intensity value for only a single color, e.g., R, GorB. A 
portion of a mosaiced image in a Bayer pattern is illustrated in Fig. 2A. 

In the exemplary embodiment, since each pixel of a mosaiced image has an 

20 intensity value for only a single color, each pixel is missing intensity values for the 
other two colors that are needed to produce a demosaiced or color image. As 
illustrated in Fig. 2A, the G-colored pixels of a mosaiced image are identified as 
either Gl or G2, which represent two different types of G-colored pixels. 
Therefore, the mosaiced image of Fig. 2 A can be decomposed with respect to four 

25 color components, R, Gl, G2 and B, as illustrated in Fig. 2B. These 

decompositions of a mosaiced image will sometimes be referred herein as Gl 
plane 202, G2 plane 204, R plane 206 and B plane 208. The Gl and G2 planes are 
collectively referred herein as the G plane. 

Turning back to Fig. 1, the image pipeline unit 104 of the image processing 

30 system 1 00 is connected to the image capturing unit 1 02 to receive the mosaiced 
images captured by the image capturing unit. The image pipeline unit operates to 
generate demosaiced and reduced images of the mosaiced images. The 
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demosaiced and reduced images will sometimes be referred herein as the final 
images. The image pipeline unit includes an image partitioning module 106, a 
statistical indicator computer 108, a color inserter 110, memory 1 12 and a 
processor 1 14. The image partitioning module 106, the statistical indicator 

5 computer 108 and the color inserter 110 represent functional blocks and are not 
necessarily separate components. These components may be embodied in the 
image pipeline unit 104 in any combination of software, firmware and hardware. 

The image partitioning module 106 of the image pipeline unit 104 operates 
to partition an input mosaiced image from the image capturing unit 102 into 

10 original image blocks. As an example, the image partitioning module may 
partition an input mosaiced image into 8x8 pixel image blocks. As described 
below, each original image block of the mosaiced image is converted into a single 
pixel to produce a final image of the input mosaiced image. Thus, the size of the 
final image, or the factor by which an input mosaiced image is reduced, depends 

15 on the size of the partitioned image blocks. 

The color inserter 1 10 of the image pipeline unit 104 operates to compute 
the R, G and B color values for the converted pixel of a final image for a given 
original image block of an input mosaiced image. The R, G and B color values of 
the converted pixel are derived from the R, Gl, G2 and B color values within a 

20 current image block of an input mosaiced image. The current image block may be 
the entire original image block of an input mosaiced image or a selected portion of 
the original image block. In the exemplary embodiment, the mean of R color 
values, the mean of G color values (both Gl and G2 color values), and the mean 
of B color values within a current image block are used as the R, G and B color 

25 values for the converted pixel of a final image. Thus, the color inserter is 
configured to compute the mean for each of the R, G and B color values of a 
current image block to provide R, G and B color values for the converted pixel of a 
final image. 

The statistical indicator computer 108 of the image pipeline unit 104 
30 operates to compute a statistical indicator for each of the R, Gl , G2 and B color 
planes of a current image block to determine the statistical distribution of color 
values within the current image block. As an example, the statistical indicator 
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may be the variance of intensity values for each color plane of a current image 
block, or other statistical indicators that can be used to measure changes of 
specific color intensity values within the current image block. The manner in 
which these statistical indicators are used by the image pipeline unit is described 
5 below. 

The processor 1 14 of the image pipeline unit 104 operates to analyze the 
statistical indicators computed by the statistical indicator computer 108 to 
determine whether the color values within a current image block are suitable to 
compute the R, G and B color values for a converted pixel of the final image. 

10 Specifically, the processor determines whether any of the computed statistical 
indicators for a current image block is greater than a predefined threshold. If so, 
the current image block is reduced by one half to produce a new current image 
block. That is, the current image block is divided by half and then one of the two 
halves is selected as the new current image block. The new current image block is 

15 used to compute new statistical indicators by the statistical indicator computer 
108. These new statistical indicators are then analyzed by the processor to 
determine whether the color values within the new current image block are 
suitable to compute the R, G and B color values for the converted pixel of the final 
image. However, if each computed statistical indicator for the current image 

20 block is not greater than the predefined threshold, the means of R, G and B color 
values of the current image block are embedded into the converted pixel of the 
final image. Embedding of color values into a pixel is defined herein as assigning 
the color values to the respective pixel. In the situation where the current image 
block includes only a single color value for the R, Gl, G2 and B color planes (i.e., 

25 the current image block is a 2x2 pixel image block), the original R and B color 
values of the current image block are used for the converted pixel of the final 
image, and the average or one of the original Gl and G2 color values is used as the 
G color value for the converted pixel. The digital representation of the converted 
pixel is then temporarily stored in the memory 1 12 of the pipeline unit 104, which 

30 may be flash or random access memory, until all the original image blocks of the 
input mosaiced image have been converted to produce the final image, i.e., the 
demosaiced and reduced image. 



Attorney Docket No. 10003433-1 



9 



The final image may be stored in the storage unit 106 and/or displayed on 
the viewing device 108 of the image processing system 100. The storage unit may 
be a conventional storage memory, such as DRAM. Alternatively, the storage unit 
may be a drive that interfaces with a removable storage medium, such as a 

5 standard computer floppy disk. The viewing device may be an LCD display or 
other comparable display that can display the final image. 

The demosaicing and resizing operation of the image processing system 
100 is described with reference to a flow diagram of Fig. 3 and the block diagram 
of Fig. 1. At step 302, an input mosaiced image of a scene of interest is received 

10 by the image pipeline unit 104. In the exemplary embodiment, the input mosaiced 
image is a Bayer patterned image captured by the image capturing unit 102. Next, 
at step 304, the mosaiced image is partitioned into image blocks. The size of the 
partitioned image blocks determines the factor by which the mosaiced image will 
be reduced. Thus, the size of the partitioned image blocks can be varied to select 

15 the size of the final image, which is the demosaiced and resized image of the input 
mosaiced image. However, in this description, the input mosaiced image is 
assumed to have been partitioned into 8x8 pixel image blocks. An exemplary 8x8 
image block is illustrated in Fig. 4A. 

Next, at step 306, one of the 8x8 image blocks is selected as a current 

20 image block to generate a pixel of the final image. At step 308, a statistical 
indicator for each of the R, Gl, G2 and B color values within the current image 
block is computed by the statistical indicator computer 108 of the image pipeline 
unit 104. In the exemplary embodiment, the statistical indicators are variances of 
the R, Gl, G2 and B color values, and thus, the statistical indicators are described 

25 herein as variances. Initially, the current image block is the selected 8x8 image 
block. However, as described below, the current image block may be a portion of 
the selected 8x8 image block. At step 310, the mean of R color values, the mean 
of G color values (both Gl and G2 color values), and the mean of B color values 
within the current image block are computed by the color inserter 1 10 of the image 

30 pipeline unit 104. Steps 308 and 3 10 may be executed in parallel. Next, at step 
312, a determination is made whether any of the computed variances exceeds a 
threshold T v , which may be empirically determined by experiments. If none of the 

Attorney Docket No. 10003433-1 



10 



computed variances exceeds the threshold, the process proceeds to step 318, where 
a pixel of the final image is generated by embedding the mean of R color values, 
the mean of G color values, and the mean of B color values for the current image 
block as the i?, G and B color values into the pixel of the final image. 

5 However, if any of the computed variances does exceed the threshold T v , 

the process proceeds to step 314, where the current image block is divided in half. 
The current image block may be divided horizontally or vertically. The manner in 
which a current image block is divided is described in more detail below. Next, at 
step 316, one of the divided image blocks is selected as the new current image 

10 block. The particular selection of the divided image block is not critical to the 
invention. After the selection of the new current image block, the process then 
proceeds back to step 308, where the new current image block is processed in the 
same manner as described above with respect to the last current image block. 
Thus, the computed means of a current image block are not used when there is 

15 significant variation in any color values, which may indicate that the current image 
block includes an edge. Consequently, the color values within such image block 
are not interpolated and used as the color values for the pixel of the final image, 
since interpolation of colors across edges may introduce undesired colors into the 
pixel of the final image and may degrade the final image. 

20 In an extreme situation, the current image block may be a 2x2 pixel image 

block, which is the smallest possible image block that still contains color 
information for each of the R, Gl, G2 and B color planes. In this situation, the 
original R and B color values of the current image block are used for the pixel of 
the final image, and the average or one of the original Gl, G2 color values is used 

25 as the G color value for the pixel of the final image. 

After step 3 1 8, the process proceeds to step 320, where a determination is 
made whether the selected 8x8 image block of the input mosaiced image is the last 
image block to be processed. If so, the process comes to an end. However, if the 
selected image block is not the last 8x8 image block of the input mosaiced image, 

30 the process proceeds back to step 306, where the next 8x8 image block of the 
mosaiced image is selected to be processed. 
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The manner in which an original 8x8 image block, such as the 8x8 image 
block of Fig. 4A, is divided one or more times in accordance with the invention is 
now described. If any of the computed variances for the 8x8 image block exceeds 
the threshold, the 8x8 image block may be vertically divided in half into two 4x8 

5 image blocks 402 and 404, as illustrated in Fig. 4B. Alternatively the 8x8 image 
block may be horizontally divided in half into two 8x4 image blocks 406 and 408, 
as illustrated in Fig. 4C. Next, if the 8x8 image block was vertically divided, one 
of the 4x8 image blocks 402 and 404 is selected as the new current image block. 
If each computed variances for the selected 4x8 image block exceeds the threshold 

10 T y , then the 4x8 image block is horizontally divided in half into two 4x4 image 
blocks 410 and 412, as illustrated in Fig. 4D. However, if the 8x8 image block 
was horizontally divided, one of the 8x4 image blocks 406 and 408 is selected as 
the new current image block. If each computed variances for the selected 8x4 
image block exceeds the threshold T v , then the 8x4 image block is vertically 

15 divided in half into two 4x4 image blocks 414 and 416, as illustrated in Fig. 4E. 
This process is continued until each computed variances for a current image block 
does not exceed the threshold T v , or until the new current image block is a 2x2 
image block, as illustrated in Fig. 4F. Since a 2x2 image block includes only a 
single value for each of the R, Gl, G2 and B color planes, the image block cannot 

20 be further divided without losing some of the color values. 

In Fig. 5, an image processing system 500 in accordance with a second 
embodiment of the invention is shown. The image processing system 500 
includes most of the components of the image processing system 100 of Fig. 1 . 
The only difference between the two image processing systems 100 and 500 is that 

25 the statistical indicator computer 1 08 of the image processing system 1 00 has been 
replaced with a feature-based indicator computer 502 in the image processing 
system 500. In contrast to the statistical indicator computer 108, the feature-based 
indicator computer 502 operates to compute a feature-based indicator, instead of a 
statistical indicator. In an exemplary embodiment, the feature-based indicator 

30 computer 502 computes the gradient for each of the R color values, G color values 
(both Gl and G2 color values), and B color values within a current image block of 
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an input mosaiced image. However, in other embodiments, the feature-based 
indicator computer 502 may compute other feature-based indicators. 

The demosaicing and resizing operation of the image processing system 
500 in accordance with the second embodiment is described with reference to a 
flow diagram of Fig. 6 and the block diagram of Fig. 1. At step 602, an input 
mosaiced image of a scene of interest is received by the image pipeline unit 104. 
In the exemplary embodiment, the input mosaiced image is a Bayer patterned 
image captured by the image capturing unit 102. Next, at step 604, the mosaiced 
image is partitioned into image blocks. Although the size of the partitioned image 
blocks can vary, the image blocks are described herein as 8x8 image blocks. 

Next, at step 606, one of the 8x8 image blocks is selected to generate a 
pixel of the final image, which is the demosaiced and resized image of the input 
mosaiced image. At step 608, a feature-based indicator for each of the R, G and B 
color values within the current image block is computed by the feature-based 
indicator computer 502 of the image pipeline unit 104. In the exemplary 
embodiment, the feature-based indicators are gradients of the R, G and B color 
values, and thus, the feature-based indicators are described herein as gradients. At 
step 610, the mean of R color values, the mean of G color values (both Gl and G2 
color values), and the mean of B color values within the current image block are 
computed by the color inserter 1 10 of the image pipeline unit 104. Steps 608 and 
610 may be executed in parallel. Next, at step 612, a determination is made 
whether any of the computed gradients exceeds a threshold T g , which may be 
empirically determined by experiments. If none of the computed gradients 
exceeds the threshold T g , the process proceeds to step 618, where a pixel of the 
final image is generated by embedding the mean of R color values, the mean of G 
color values, and the mean of B color values for the current image block as the R, 
G and B color values into the pixel of the final image. 

However, if any of the computed gradients do exceed the threshold T g , the 
process proceeds to step 614, where the current image block is divided in half. 
The current image block may be divided horizontally or vertically. Next, at step 
616, one of the divided image blocks is selected as the new current image block. 
The process then proceeds back to step 608, where the new current image block is 
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processed in the same manner as described above with respect to the last current 
image block. Thus, the computed means of a current image block are not used 
when there is a significant gradient with respect to any of the color values within 
the current image block, which indicates that the color values of the current image 
5 block are not suitable to generate interpolated color values for the pixel of the final 
image. Consequently, the current image block is divided to search for an image 
block that has color values suitable for the pixel of the final image. 

After step 618, the process proceeds to step 620, where a determination is 
made whether the selected 8x8 image block of the input mosaiced image is the last 

10 image block to be processed. If so, the process comes to an end. However, if the 
selected image block is not the last 8x8 image block of the input mosaiced image, 
the process proceeds back to step 606, where the next 8x8 image block of the 
input mosaiced image is selected to be processed. 

The foregoing descriptions of specific embodiments of the invention have 

15 been presented for the purposes of illustration and description. These descriptions 
are not intended to be exhaustive or to limit the invention to the disclosed 
embodiments disclosed. Many modifications and variations are possible in light 
of the above teaching. The embodiments were chosen and described in order to 
explain the principles of the invention and its practical application, to thereby 

20 enable others skilled in the art to best utilize the invention and embodiments with 
various modifications as are suited to the particular use contemplated. It is 
intended that the scope of the invention be defined by the Claims appended hereto 
and their equivalents. 
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